Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning

Real-world cooperation often requires intensive coordination among agents simultaneously. This task has been extensively studied within the framework of cooperative multi-agent reinforcement learning (MARL), and value decomposition methods are among those cutting-edge solutions. However, traditional methods that learn the value function as a monotonic mixing of per-agent utilities cannot solve the tasks with non-monotonic returns. This hinders their application in generic scenarios. Recent methods tackle this problem from the perspective of implicit credit assignment by learning value functions with complete expressiveness or using additional structures to improve cooperation. However, they are either difficult to learn due to large joint action spaces or insufficient to capture the complicated interactions among agents which are essential to solving tasks with non-monotonic returns. To address these problems, we propose a novel explicit credit assignment method to address the non-monotonic problem. Our method, Adaptive Value decomposition with Greedy Marginal contribution (AVGM), is based on an adaptive value decomposition that learns the cooperative value of a group of dynamically changing agents. We first illustrate that the proposed value decomposition can consider the complicated interactions among agents and is feasible to learn in large-scale scenarios. Then, our method uses a greedy marginal contribution computed from the value decomposition as an individual credit to incentivize agents to learn the optimal cooperative policy. We further extend the module with an action encoder to guarantee the linear time complexity for computing the greedy marginal contribution. Experimental results demonstrate that our method achieves significant performance improvements in several non-monotonic domains.Comment: This paper is accepted by aamas 202

A protocol for rapid construction of senescent cells

Aging may be the largest factor for a variety of chronic diseases that influence survival, independence, and wellbeing. Evidence suggests that aging could be thought of as the modifiable risk factor to delay or alleviate age-related conditions as a group by regulating essential aging mechanisms. One such mechanism is cellular senescence, which is a special form of most cells that are present as permanent cell cycle arrest, apoptosis resistance, expression of anti-proliferative molecules, acquisition of pro-inflammatory, senescence-associated secretory phenotype (SASP), and others. Most cells cultured in vitro or in vivo may undergo cellular senescence after accruing a set number of cell divisions or provoked by excessive endogenous and exogenous stress or damage. Senescent cells occur throughout life and play a vital role in various physiological and pathological processes such as embryogenesis, wound healing, host immunity, and tumor suppression. In contrast to the beneficial senescent processes, the accumulation of senescent also has deleterious effects. These non-proliferating cells lead to the decrease of regenerative potential or functions of tissues, inflammation, and other aging-associated diseases because of the change of tissue microenvironment with the acquisition of SASP. While it is understood that age-related diseases occur at the cellular level from the cellular senescence, the mechanisms of cellular senescence in age-related disease progression remain largely unknown. Simplified and rapid models such as in vitro models of the cellular senescence are critically needed to deconvolute mechanisms of age-related diseases. Here, we obtained replicative senescent L02 hepatocytes by culturing the cells for 20 weeks. Then, the conditioned medium containing supernatant from replicative senescent L02 hepatocytes was used to induce cellular senescence, which could rapidly induce hepatocytes into senescence. In addition, different methods were used to validate senescence, including senescence-associated β-galactosidase (SA-β-gal), the rate of DNA synthesis using 5-ethynyl-2′-deoxyuridine (EdU) incorporation assay, and senescence-related proteins. At last, we provide example results and discuss further applications of the protocol

IRGen: Generative Modeling for Image Retrieval

While generative modeling has been ubiquitous in natural language processing and computer vision, its application to image retrieval remains unexplored. In this paper, we recast image retrieval as a form of generative modeling by employing a sequence-to-sequence model, contributing to the current unified theme. Our framework, IRGen, is a unified model that enables end-to-end differentiable search, thus achieving superior performance thanks to direct optimization. While developing IRGen we tackle the key technical challenge of converting an image into quite a short sequence of semantic units in order to enable efficient and effective retrieval. Empirical experiments demonstrate that our model yields significant improvement over three commonly used benchmarks, for example, 22.9\% higher than the best baseline method in precision@10 on In-shop dataset with comparable recall@10 score

Model-enhanced Vector Index

Embedding-based retrieval methods construct vector indices to search for document representations that are most similar to the query representations. They are widely used in document retrieval due to low latency and decent recall performance. Recent research indicates that deep retrieval solutions offer better model quality, but are hindered by unacceptable serving latency and the inability to support document updates. In this paper, we aim to enhance the vector index with end-to-end deep generative models, leveraging the differentiable advantages of deep retrieval models while maintaining desirable serving efficiency. We propose Model-enhanced Vector Index (MEVI), a differentiable model-enhanced index empowered by a twin-tower representation model. MEVI leverages a Residual Quantization (RQ) codebook to bridge the sequence-to-sequence deep retrieval and embedding-based models. To substantially reduce the inference time, instead of decoding the unique document ids in long sequential steps, we first generate some semantic virtual cluster ids of candidate documents in a small number of steps, and then leverage the well-adapted embedding vectors to further perform a fine-grained search for the relevant documents in the candidate virtual clusters. We empirically show that our model achieves better performance on the commonly used academic benchmarks MSMARCO Passage and Natural Questions, with comparable serving latency to dense retrieval solutions

Relationship between urban heat island and green infrastructure fraction in Harbin

Urbanisation contributed to the presence of urban heat island phenomenon, and aggravated urban heat island effect intensity with the improvement of urbanisation level. Overheat weather condition caused severe threat to human life and health, while green infrastructure including water bodies has been validated to be able to reduce urban land surface temperature in different extent. To examine the impact of green infrastructure on urban heat island effect in Harbin, with the aid of ENVI and geographic information system software, this paper retrieved seasonal Harbin land surface temperature from 2000 to 2015 using Landsat series and MODIS 8-day remote sensing data, and further computed surface urban heat island intensity(SUHII). Then, to build the quantitative relationship between green infrastructure fraction and urban heat island intensity applying regression analysis method. Finally, by means of ENVI-MET software, this article simulated urban heat island intensity change based on different green infrastructure scenarios. The results showed that, as far as administrative region of Harbin scale, surface urban heat island intensity both in summer and in winter reduced from 2000(6.55°C in summer, 4.15°C in winter) to 2015(2.6°C in summer, 0.47°C in winter), and SUHII in summer is higher than it in winter except 2005; Green infrastructure fraction is negative correlated with SUHII; Simulation result indicated that increase on green infrastructure would facilitate to mitigation of urban heat island effect. The result of this study would provide some help and advice for land use planning decision and urban construction in the future of Harbin

Recent advances of PROTACs technology in neurodegenerative diseases

Neurodegenerative diseases, like Alzheimer's disease, Huntington's disease, Parkinson's disease, progressive supranuclear palsy, and frontotemporal dementia are among the refractory diseases that lack appropriate drugs and treatments. Numerous disease-causing proteins in neurodegenerative diseases are undruggable for traditional drugs. Many clinical studies of drugs for Alzheimer's disease have failed, and none of the substances that slowed the amyloid-β (Aβ) accumulation process have been approved for use in clinical trials. A novel approach to addressing this issue is Proteolysis targeting chimeras (PROTACs) technology. PROTACs are heterogeneous functional molecules joined by a chemical linker and include binding ligands for the target protein and recruitment ligands for the E3 ligand. When a PROTAC binds to a target protein, the E3 ligand enzyme is brought into close contact and the target protein begins to be polyubiquitinated, followed by proteasome-mediated degradation. Numerous neurodegenerative disease-related targets, including α-Synuclein, mHTT, GSK-3, LRRK2, Tau, TRKA, and TRKC have been successfully targeted by PROTACs to date. This article presents a comprehensive overview of the development of PROTACs in neurodegenerative diseases. These PROTACs' chemical structures, preparative routes, in vitro and in vivo activities, and pharmacodynamics are outlined. We also offer our viewpoint on PROTACs' probable challenges and future prospects